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Abstract 

In this article, we describe the stages of a formative evaluation process for a professional 
development project. The professional development project’s focus was to support teachers 
in urban schools to implement student- centered practices. The purpose of this article is to 
highlight what we learned about formative evaluation and the importance of formative 
evaluation in shaping the professional development project. 
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Formative Evaluation of Professional Development: Hcrw mil we knowsuccess? 

With today’s focus on outcomes and accountability, those of us who provide 
professional development are often asked for summative evaluation (i.e., “did it work?”) 
done by an outside evaluator at the end of a project. Yet, it is often more useful to engage in 
formative evaluation as a way to not only shape our work but also to help us attribute 
desired outcomes to our work 

In this article, we describe what we learned from a formative evaluation project in 
which an outside evaluator and program staff collaborated (see Figure 1 for a description of 
this project). We have conceptualized four “stages,” or conceptual periods, which serve 
both as a “road-map” for this type of collaboration and the organizing structure for this 
article. Table 1 highlights the key focus questions for each stage. 

Stage 1: What is the program usum and ishat are the anticipated changes in practice? 

An essential stage in evaluating a program is defining it so that linkages between 
program inputs and outcomes can be made. To do this, we must essentially spell out the 
program’s key assumptions, moving beyond the general goals typically articulated at the 
outset of a project. For us, an important strategy was to have program staff define "success" 
(i.e., program outcomes). Then, starting with these indicators of success, we worked to link 
them to what should be specific program inputs. This process included defining the 
program’s philosophical approach to working with teachers. Connell and Klem (2000) refer 
to this process as articulating a program’s “theory of change.” 

Although this sounds simple, we learned it is a complex and thought- provoking 
process involving an interplay of sometimes competing goals. For example, program staff 
noted the importance of meeting "teachers where they are.” Just as program staff felt that 
teachers should meet individual student needs, they felt it was important for themselves to 
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meet individual teacher needs. However, such an orientation can be difficult for evaluation 
research where “fidelity of implementation” is critical for making any conclusions about 
program effectiveness. Thus, the evaluator’s role was to focus program staff on clarifying 
the program to move them beyond the arena of individual teachers. 

Another focus area during this stage was designing outcome measures. One 
important measure was an observation tool to assess level of implementation by teachers, 
which could be used to evaluate the professional development activities. Later, this 
information would be important for linking teachers who implemented the key instructional 
practices to student learning outcomes. A significant challenge for program staff was to 
conceptualize essential instructional features. Our dialogues often focused on the question, 
"What are the core practices that we would see in a student centered classroom?" 

Providing uniform consultation and coaching to the teachers also posed a challenge. 
One lengthy project discussion focused on data that indicated high variability amongst 
program staff on their provision of feedback and suggestions. For example, one program 
staff tended to focus on classroom climate issues; another focused on reading strategies; and 
another focused primarily on scheduling and organizational structures within the classroom. 
Each program staff appeared to focus on areas in which they had the most expertise. There 
was also variability on how much staff "pushed" teachers to implement strategies from the 
observation measure. Given that the success of the program would be based on this 
instrument, program staff agreed to focus coaching sessions towards increasing levels of 
implementation on these practices and to make the process more uniform. 

Stag? 2: Who are our teacher learners and hcrware they responding to the professional development program? 

Just as teachers need to know their students to be effective in their teaching, 
program staff needed to understand the participants in their program. This is generally a 
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cornerstone of professional development efforts (Guskey, 1997), and for this reason, many 
programs gather pertinent background information on their participants. We know that by 
approaching teacher participants as unique individuals with a diverse range of attitudes, 
concerns, and skill levels, the content and structure of the program can better meet teachers’ 
needs. We were also interested in understanding for whom the program would likely be 
successful, given the resources that we had and our need to fairly quickly link impacts to 
teachers’ students. 

Based on analysis of teacher- completed Stages of Concern surveys (Loucks-Horsley, 
Newlove, & Hall, 1975), which identifies teachers' concerns when innovations are 
introduced, we learned that our teacher participants were mostly concerned with 
informational, management, and collaboration needs. As a result, program staff focused on 
addressing these concern areas. In addition, we analyzed responses on the Reflection on 
Teaching Scale (Gibson &Dembo, 1984), which measures the level of teacher efficacy. Data 
from this scale helped us to focus on building teachers’ sense of efficacy where indicated, 
because research tells us that teaching efficacy can be an important contributor to whether 
teachers implement innovative teaching practices (Fritz & Miller- Heyl, 1995; Guskey, 1988). 
Later, we wanted to see if this information would be relevant for helping us identify for 
whom the program would be successful, or whether it would be an important teacher 
outcome measure that would link to positive student outcomes. 

Another way we could assess whether teachers were likely to implement specific 
practices was through a survey that teachers completed following each workshop. Teachers 
were asked to reflect on the training workshop in four categories: 1) whether they 
understood the strategy presented; 2) whether they viewed the specific strategy as being 
important; 3) whether the strategy appeared to "fit" for their classroom situation; and 4) their 
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plans for implementing the strategy. With feedback across these four dimensions, we were 
able to pinpoint barriers to implementation and respond more appropriately to teachers’ 
needs. 

Stag; 3: What is the impact on teachers’ students ? 

Discussions amongst program staff led us to wonder how we could be certain that 
student outcomes could in fact be due to our professional development program. This is 
particularly complex, since many schools currently are engaged in a myriad of activities that 
could have potential impacts on student outcomes. Given our “theory of change” that was 
developed as part of Stage 1, we felt that there should be a linkage between teachers’ level of 
implementation and student outcomes. We believed that if teachers were implementing the 
practices, then their students should have more positive outcomes than the teachers who 
were not implementing the practices. This guided our subsequent statistical analyses, and 
emerging findings indicated that the lower performing students actually showed the most 
gains if they were with a teacher who was a “high implementor.” 

Stag 4: What doangs need to be made to the professional deidopment program- 

Because this was a formative evaluation project, we were also concerned with using 
the information gathered to inform future programs. Based on our examination of all the 
data, this stage was characterized as a time in which the program features were re-examined, 
and some areas were dropped while others were given more attention. For example, based 
on the teacher data, it was clear that most still needed support to use student assessments 
and assessment data to inform their teaching. In addition, program staff realized that the 
content covered during the first year should be revisited during subsequent workshops so 
that the concepts could be solidified for teachers. 
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Conclusion and Implications 

Reflection on the current evaluation project led to the creation of a “road-map” that 
is useful for helping us understand the key process or stage in which we are engaged. We 
believe that developing this common framework between program evaluators and program 
developers is critical to a useable evaluation. In addition, we believe the following lessons 
are important for practice, particularly for those who fund professional development 
programs, those who design them, and those who evaluate them. 

E ariy development of an obsermtim tool can help to focus a staff development program 

Especially for program developers who are seeking to work with an evaluator, the 
process involved in creating an evaluation tool actually helps to refine and specify the 
program’s goals and visions. In other words, the process itself helps to clarify what the 
program seeks to accomplish. Further, creation of an observation tool forced us to 
articulate a specific practice in observable terms. As Morris and Fitz-Gibbon (1985) note, 
"You cannot evaluate something without describing what that something is" (p. 8). 

It is essential to have adequate time before examining the inpact on participants’ students. 

Professional development programs are increasingly being asked to show the impact 
of their programs on student achievement. Funders and school district sponsors of 
professional development are anxious to see gains by the end of the first or second year of 
training. Staff developers are often put in the position of making claims that particular 
innovations will increase student achievement before it is known whether the trai nin g 
provided resulted in a change in practice, or whether there has been sufficient change in 
practice to have an impact on student learning. Unfortunately, this may lead to erroneous 
conclusions about a particular professional development program. Some programs might 
indicate results in improved student learning, despite the fact that few teachers actually 
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implemented the intended practices. Still, another program might indicate lack of improved 
student learning, however, the program may actually be quite successful in improving 
student learning but not enough teachers implemented the practices to a sufficient degree. 

In our experience, high levels of implementation could reasonably take more than two years 
to develop. 

Data collected can help ns answer for whom and under uhat conditions the staff deudopment program works. 

One important outcome of a formative evaluation process is that we can begin to 
explore one critical question: “For whom does the program development program work, and 
under what conditions?” Those responsible for professional development programs are 
often interested in this question, given the limited resources that many programs face. In 
this project, we collected a variety of information on the teacher learners that was then used 
to extend our understanding of teachers who subsequently utilized the practices to a fairly 
high level of implementation. Some evaluation projects refer to this as “readiness 
indicators.” Unfortunately, programs rarely collect the kind of information that can answer 
the question “for whom and under what conditions?” Consequently, programs may find 
themselves being less efficient with the available resources. We have found that we need to 
move beyond “buy-in” criteria that are typically based on teachers agreeing to be part of a 
professional development program. In our experience, there are many reasons that teachers 
may agree to participate in a professional development program, and not all reasons 
necessarily result in participants who will implement new practices to a high degree. 
Unfortunately, in times of scarce resources, we will need a way to decide where best to put 
our efforts and resources. 
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Fomntiwcdlabcratimbetaiem pro^am dexdopers and an outside euduator serves to get the program on 
track early, and keep it on track. 

Formative evaluation that begins at the outset of a professional development 
program can help sharpen program objectives and strategies, which leads to maximizing the 
potential benefit of the program to teachers and their students. Program staff have a vested 
interest in the success of their work as they have designed it and can be insensitive to needed 
changes. The importance of another set of eyes provided by a program evaluator is 
described by Morris and Fitz-Gibbons (1985): 

The formative evaluator's job is not only to describe the program, but also to keep a 
vigilant watch over its development and to call the attention of the program staff to 
what is happening. Program monitoring in formative evaluation should reveal to 
what extent the program as implemented matches what its planners intended, and 
should provide a basis for deciding whether parts of the program ought to be 
improved, replaced, or augmented, (p. 14) 

Thus, formative evaluation not only helps define the professional development program’s 
goals and methodology, it ensures that implementation is consistent and that student 
outcomes can be attributed to the use of classroom strategies promoted by the program. 
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Figure 1. Description of the Formative Evaluation Project and the Professional 
Development Program 



The Student Centered Learning Project was designed to train and support urban elementary 
school teachers to provide instructional approaches and learning environments leading to 
increased student engagement and academic success. The current project focused on assisting 
teachers with creating a positive learning climate and with implementing effective literacy 
instruction for struggling readers. Originally designed as a professional development program for 
primary level elementary school teachers, the current program was conceptualized to extrapolate 
the key components of student-centered learning approaches for upper elementary school 
teachers. The program included monthly workshops along with follow-up individual coaching 
sessions provided by program staff. 

This project was funded by local foundations that were interested in improving the learning 
outcomes of students in urban schools. These funding agencies requested that an evaluation 
component be included in order to evaluate the impact of the program on teachers and students. 
The project was funded as a two-year pilot program, during which time, the goal was to both 
develop an evaluation approach and provide professional development to at least twenty 
classroom teachers who were teaching in urban school settings. The formative evaluation project 
included monthly debriefs between an external program evaluator and program staff. These 
debriefs included program status reports, in-depth discussion of programmatic and evaluation 
issues, and co-development of evaluation instruments. 
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Table 1. Focus Questions for Each Stage of the Formative Evaluation Process 



Stage 1 : What is the 


v What are the critical features of the professional 


program vision and 


development program? 


what are the resulting 


v How will the program work with teacher participants? 


changes in practice? 


s How will we know if the program is successful? 


Stage 2: Who are the 


s What are important teacher characteristics (i.e., efficacy, age, 


teacher learners, and 


educational background, prior experiences, teaching 


how are they 


philosophy, etc.) that might influence implementation? 


responding to the 


v What are important contextual factors (i.e., student 


professional 


demographics, teaching assignment, etc.) that might 


development program? 


influence implementation? 

v How will we know which teachers are implementing the 
instructional practices? 


Stage 3: What is the 


v In what areas of academic development do we expect to see 


impact on teachers’ 


impacts on students? 


students? 


v What are appropriate student measures? 
v How can we be sure that the impact on students can be 
attributed to the professional development program? 


Stage 4: What changes 


v What aspects of the program need to be changed, dropped, 


need to be made to the 


or refined? 


professional 




development program? 
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